Add initial benchmarking setup. #892
Conversation
jerzywilczek
requested review from
wkozyra95,
noituri,
brzep,
WojciechBarczynski and
wkazmierczak
jerzywilczek
force-pushed
the
@jerzywilczek/initial-benchmark
branch
6 times, most recently
from
4e52a6c to
131428a
Compare
| fn run_args_iterate( | ||
| ctx: GraphicsContext, | ||
| args: &Args, | ||
| arguments: Box<[Argument]>, | ||
| reports: &mut Vec<SingleBenchConfig>, | ||
| ) -> bool { | ||
| for (i, argument) in arguments.iter().enumerate() { | ||
| if matches!(argument, Argument::IterateExp) { | ||
| let mut any_succeeded = false; | ||
| let mut count = 1; | ||
| loop { | ||
| let mut arguments = arguments.clone(); | ||
| arguments[i] = Argument::Constant(count); | ||
|
|
||
| if run_args_iterate(ctx.clone(), args, arguments, reports) { | ||
| any_succeeded = true; | ||
| count *= 2; | ||
| continue; | ||
| } else { | ||
| return any_succeeded; | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| run_args_maximize(ctx, args, arguments, reports) | ||
| } |
There was a problem hiding this comment.
I'm not sure about this code. On the first sight I thought there might be an infinite recursion. While now I understand what's happening here, the flow here is hard to follow. Also, I tried to run --decoder-count with iterate_exp and it didn't do anything. I'm not sure if I used it correctly tho
There was a problem hiding this comment.
you must have not set any argument to maximize, at which point the test is kind of pointless. This should be caught and reported as a problem: there's nothing you can learn by just setting all arguments to iterate and none to maximize. If you set framerate to iterate_exp and decoders to maximize it will work.
I'll add code to catch this sort of a mistake.
There was a problem hiding this comment.
I also added comments in some places, because I don't have ideas on how to make this easier to read. I also changed the return type of this function to struct FurtherIterationPossible(bool) to make it clear what the returned bool means.
jerzywilczek
force-pushed
the
@jerzywilczek/initial-benchmark
branch
from
131428a to
eda531f
Compare
jerzywilczek
force-pushed
the
@jerzywilczek/initial-benchmark
branch
from
eda531f to
e7c6538
Compare
Here's an initial benchmarking setup. For context, the end goal for this is for us to be able to run a set of these benchmarks on any computer and figure out what kind of smelter setups can be run on it.
You can run it with:
to get the information on how to run a single test.
The full command is quite verbose, and looks like this:
cargo run --release --bin benchmark -- \ --framerate 24 \ --decoder-count maximize \ --file-path examples/assets/BigBuckBunny.mp4 \ --output-width 1280 \ --output-height 720 \ --encoder-preset ultrafast \ --warm-up-time 10 \ --measured-time 10 \ --video-decoder vulkan_video_h264 \ --framerate-tolerance 1.05The help message will tell you all of the options for each of the arguments.
It can currently run a single test configuration. A single configuration is comprised of a couple of constant parameters (the resolution, the input video, the encoder preset etc.) and currently two variable parameters: framerate and decoders count. The variable parameters can be set to:
Importantly, you can set multiple parameters to iterate and a single parameter to maximize, which would mean the setup would run for all iterative combinations and for each of them find the maximum number for the maximized parameter.
For example, if we set the framerate to iterate and the decoders count to maximize, the setup will run with framerate set to 1, 2, 4, 8, 16 etc. and for each framerate find the maximum number of inputs. The iteration will terminate when it is impossible to run the system due to the super high framerate even with a single decoder.
What I plan to add to this later:
Additionally to asking whether you like this system or think it should be changed somehow, I would like to get your opinion on my handling of the arguments that are set to iterate or maximize (the
run_argsfunction and the other functions it calls). I feel a bit like using zig for the last couple weeks got into my head and that there is a better, more idiomatic way of doing what I'm doing here, but I just can't figure it out.I also fixed an unrelated typo in vk-video, because it was annoying me.